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DESCRIPTION 

Method and Apparatus for Face Description and Recognition 
using High-order Eigencomponents 

5 

Technical Field 

The present invention relates to method and apparatus 
for face description and recognition using High-order 
Eigencomponents. The invention can be used in face 
10 description and recognition for content-based image 
retrieval, human face identification and verification for bank, 
security system and videophone; surveillance and tracking; 
digital library and Internet multimedia database. 

15 Background Art 

Human face perception is an active area in the 
computer vision community. Face recognition will play an 
important role in multimedia database search and many 
other applications. In recent years considerable progress 

20 has been made on the problems of face detection and 
recognition. Different techniques were proposed. Among 
these, neural networks, elastic template matching, 
Karhunen-Loeve expansion, algebraic moments and 
isodensity lines are typical methods. 

25 Among these methods, principal component analysis 



wo 03/088131 




[CT/JP03/04550 



2 



(PCA) or Karhunen-Loeve expansion is an important branch. 
Elgenface method is derived from PCA and it is convenient 
to be computed and has consistent accuracy in 
identification. Prior work showed that the PCA approach 
5 dissociates spontaneously between different types of 
information. The eigenvectors with large eigenvalues 
capture information that is common to subsets of faces and 
eigenvectors with small eigenvalues capture information 
specific to individual face. The studies show that only the 
10 information contained by the eigenvectors with large 
eigenvalues can be generalized to new faces that are not 
trained. 

The advantage of eigenface method is that 
eigenvectors with large eigenvalues convey information 

15 relative to the basic shape and structure of the faces. That 
means features extracted from eigenvectors with large 
eigenvalues can be used to describe major characteristics 
of human faces. However, this is also the weakness of PCA. 
If we only consider the features extracted from 

20 eigenvectors with large eigenvalues, we cannot get the 
details of faces which corresponds to individual face. If 
these details of individual face can be described with the 
common features of human faces, the description of human 
faces can be more accurate. 

25 One drawback of eigenface method is that the 
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contributions of all facial component areas are same. 
Instead of evenly distributed on the whole face, the identity 
information mainly locates at the certain facial areas, such 
as eyes, eyebrows, nose, mouth and outline. The check 
5 area contains less identity information and is relatively 
sensitive to lighting condition changes and facial 
expression changes. If the identity significances of facial 
components can be used, the recognition of human face can 
be more accurate. 

10 Disclosure of Invention 

Eigenface method is effective to extract common face 
characteristics such as shape and structure, in order to get 
details of faces that are lost when eigenvectors with small 
eigenvalues are truncated, the reconstructed faces with the 

15 features from eigenvectors with large eigenvalues should 
be obtained. With the reconstructed face images, the 
residue images between original images and reconstructed 
matrices can be obtained. These residue faces can be 
looked as high-passed face images, which still contain rich 

20 detailed information for individual face. In order to describe 
these residue faces, eigenface method can be used on 
these residue faces again. The obtained eigenvectors with 
large eigenvalues will reveal the common characteristics of 
residue faces. With this method, high-order eigenvectors 

25 with large eigenvalues can be obtained to extract 
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corresponding features. The combination of these features 
from different order eigenfaces can be used to describe 
faces effectively. 

Similarly, first order and higher order principal 
5 components (eigencomponents) of facial components can be 
obtained to describe the characteristics of the 
corresponding facial areas. The combination of these 
features from different order eigencomponents can be used 
to describe individual facial components efficiently. Finally, 

10 human faces can be represented by a combination of 
different order eigencomponents with different attention 
weights. For different application fields, different 
components should have different functionalities (its 
strongerness or weakness). Different weights should be 

15 assigned for that component. 

The present Invention provides a method to interpret 
human faces which can be used for image retrieval (query 
by face example), person identification and verification, 
surveillance and tracking, and other face recognition 

20 applications. In order to describe face characteristics, the 
concept of high-order eigencomponents is proposed 
according to our observation and derivation. At first, all 
face images are normalized to a standard size. Then the 
vertical location of eyes is calculated and the face is 

25 shifted to a suitable place. When all these pre-processing 
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procedures are finished, the eigencomponents and high- 
order eigencomponents can be derived from a set of 
training face images. In order to query a face image in a 
face database, the features of the image projected with 
5 eigencomponents and high-order eigencomponents can be 
calculated with the selected eigencomponents and high- 
order eigencomponents. The combination of these features 
can be used to describe faces. With this description, 
Euclidean distance can be used for similarity measurement. 
10 In order to improve the similarity accuracy, the features 
should be weighted. 

Brief Description of Drawings 

Figure 1 shows a flowchart of the procedure for 

15 computing first -order feature 

Figure 2 shows a flowchart of the procedures for 

computing i'^-order eigencomponents ^^'^ and the 

corresponding transform matrix 

Figure 3 shows a flowchart for the training mode 

20 operation. 

Figure 4 shows a flowchart for the test mode operation. 

Best Mode for Carrying Out the Invention 

The present invention gives a method to extract higher 
25 order elgencomponent features and represent a face by 
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combining different order component features. 

With the normalized face images, the eigencomponents 
and high-order eigencomponents can be obtained as follows. 

First use a preset block for the normalized face 
5 Images to get the facial component (such as eyes, 
eyebrows, nose, mouth and outline); 

Consider a facial component which is a one- 

dimensional vector of raster-scanned facial component, 
define ^as the average component: 

1 M 

10 4' = -i-s^i (1) 

Every facial component differs from the average 
component by a vector 

r/'^=0,-Y. (2) 
The covariance matrix of the data is thus defined as: 

15 Q^AP^^A'^^'' (3) 

where 

^c^) = [r/V^..r^<^>]. (4) 

20 Note that Q has dimension whxwh where w is the width of 
the component and h is the height. The size of this matrix 
is enormous, but since we only sum up a finite number of 
component vectors M, the rank of this matrix can not 
exceed M-1 . We note that if v.^Ms the eigenvector of ^^'^''^4^'^ 
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(i = 1.2 M), then ^^'^''^^V'^ = where are the 

eigenvalue of A^^^A^^\ then are the eigenvectors of 

^o)^o)^as we see by multiplying on the left by A^^^ in the 
previous equation: 

5 

^o)^o)^^(i)^^o) = ^0)^0^0) = ;i,«^('>v,™ ( 5 ) 

Thus, eigenvector v,^^ and eigenvalue A/*^ are obtained 
by the following equation. 

10 

[v«;j,^'^]=eig(^<'^^«0, i = l.••M^ (6) 

But A^^^^A^^^is only of size MxM. So defining w/'Hhe 
eigenvector of ^^'M^'^^we have 

15 

«o> = ^ov> = |;v/V> (7) 

The eigenvalue is the variance along the new 

coordinate space spanned by eigenvectors m,^\ From here 
on we assume that the order of / is such that the 
20 eigenvalues A/'^are decreasing. The eigenvalues are 
decreasing in exponential fashion. Therefore we can project 
a facial component r^^onto only M^«M dimensions by 
computing FF^'^ = K^'^} where w/^ = «/^''r^'^ and l^k<.M,. w,^'^ is 
the k'th coordinate of I^^^in the new coordinate system. In 
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this context, W^^^ is called first -order component features. 
The vectors if/^^are actually images, and are called first- 
order eigencomponents. 
Let 

5 C/^^^=[i^,V...i/^,^^]. (8) 

then 

Since U^^^\s an M^xPmatrix, we cannot get its inverse. 
However, we can use its pseudo inverse to approximate its 

10 inverse. Let U^^^^be the pseudo-inverse of U^^^^ as indicated 
below, 

U^'^ = Pseudo - JnviU^'^) (10) 

15 then 

where f^'^ is the reconstructed matrix from FT^^and U^^K 
Then, the following equation is carried out to obtain a 
residue component r/^\ 

20 

Since a residue facial component vector still contains rich 
information for the individual component, facial component 
25 features should be extracted from the residue component 
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again. Let ^^'> == [r/'^r^^'^.Tj^^^'^J, >ip>be the eigenvalues of 

and vp^ be the corresponding eigenvectors of A^^^^A^'^^. Then 

^(2/^(2)^(2) _ ^(2)^(2) Based on above discussion, the 

eigenvectors of -4^^^-4^^^^are uj^^-A^^\f^, Therefore we can 
project a residue component r^^^onto only M^«M 
dimensions by computing W^'^^ where 

(13) 

and \<.k<.M^. Since u^^^ are the eigenvectors of the residue 
component, we call u^'^^ the second -order eigencomponents 
and Wjt^^^ the second -order component features. 



Let 

£/^^^ = kV^-V'1« (14) 
Eq.(13) can be written as 

= (t/^^^^ ^u^^y'v^'^^u^'^^'y^'^ (15) 

Let 

= (c/(2)^ - t/w^f/coV^/jF , (16) 

we have 

fr^">=t//r^>, (17) 

Since is a constant transform matrix and it is just 
calculated once, it will not affect the efficiency of 
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computation. The facial component can be described with 

a(o)= ^/^H..^^...,w^./'\^/^\<\....^^/^>f . (18) 

5 where 1<M,*^M,. The computational burden does not 
increase in computing compared with only computing 

component features from eigencomponents U. 

The residue components are called second -order 
residue components and the original components are called 

10 first -order residue components. 

With the same method, we also can derive third-order, 
fourth-order , and n*^-order eigencomponents. By 
projecting the residue components of corresponding order, 
we can get third-order, fourth-order n*-order 

15 component features. With these high-order component 
features, the similarity of components can be defined as the 
weighted Euclidean distance between the projections. Fig. 
2 illustrates the procedure to compute the i^^'-order 
eigencomponents U^'^ and the corresponding transform 

20 matrix C/,. In the figure, Pseudo_lnv(B) Is the function to 

calculate the pseudo-inverse of matrix B. 

The measure of dissimilarity of two faces and H2 is 
defined as a combined distance between various facial 
component features generated from the projections of 
25 eigencomponents (i.e. eigeneyes, eigeneyebrows, 



wo 03/088131 ^jjj^ ^^T/JP03/04550 

11 



eigennoses, eigenmouths and eigenoutlines) and 
eigenfaces. 



i=l /si 

j^t^^brow eyebrow 
now now 
tnouth mouth 



If a,=0, the similarity of face images will be measured 
only with second -order features. 

With the method mentioned above, the method for 
describing a face image can be considered as follows: 
1) Scanning the said facial component by using a raster 
scan starting at the top-left corner of the component 
window and finishing at the bottom-right corner of the 
component window into a single dimensional array of pixels; 
2} Subtracting the said single dimensional array of pixels 
with the average component; 

3) Multiplying the said subtracted single dimensional 
array of pixels with the said first-order and high-order 
eig en components; 
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4) Using the resulting component features as description 
of the face; 

5) Coding the said features into a coded representation. 
With this method, the human faces can be described 

5 effectively and efficiently in space with different attention 
weights corresponding to the significance of identity 
Information of human face. 

Next, an overall operation of the fadal feature extraction according to the 
present invention is described with reference to Figs. 3 and 4. The operation 
1 0 includes a training mode operation (steps #22-#31) as shown in Fig. 4, and a 
test mode operation (steps #32h#42) as shown In Fig. 3. The training mode 
operation is carried out first to learn and accumulate many face samples and to 
obtain coefficients of a first order average. 

The training mode starts from step #22 and continues to step #31 . The 
15 training mode is provided to generate various parameters to be used in the test 
mode. 

In step #22, a plurality of sample face images are Input. 
In step #23, each sample face image is divided into a plurality of facial 
parts, such as right eye. left eye, right eyebrow, left eyebrow, nose, mouth, 
20 facial configuration, and each part is analyzed to obtain basic facial component 
®, . The facial component % can be weighted according to the significance of 
identity of human face. 

In step #24, the facial components of the same part, such as a nose part, 
are collected from a plurality of sample face images. The facial component of 
25 nose part is referred to as a nose component. The collected facial components 
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of each part are averaged to obtain a first order average facial component ^ by 
using equation (1). The same operation is canried out to obtain the first order 
average facial components for different facial components. The first order 
average facial component ^ is used in the test mode operation in step #34. 
5 In step #25, the facial component of, for example, nose from each 

sample face image is subtracted by the first order average facial component 
^ of nose to obtain a vector of nose given by equation (2). The same operation 
is carried out for each of different facial components. 

Steps #22 to #25 taken together is called an analyzing step for analyzing 
1 0 the training face images. 

In step #26, equations (4), (5), (6), (7) and (8) are carried out to obtain 
the first order eigencomponents U^^'^ . The first order eigencomponents U^^^ is 
used in the test mode operation in step #35. 

In step #27, an inverse matrix U^^^ is generated by using equation (10). 
15 The inverse matrix U^^^ is nearly equal to C/^^^ . The inverse matrix U^^^ is 
used In the test mode operation In step #37. 

In step #28, the inverse matrix U^^^ is used for obtaining a difference of 
the facial component using equation (11). Thus, a difference of facial 
component with respect to the basic facial component of the original facial 
20 images as collected in step #22 is obtained. The data produced in step #28 is 
referred to as a difference facial component. 

In step #29, a difference between the difference facial component and 
the first order average facial component is obtained. 

Steps #27 to #29 taken together is called an analyzing step for analyzing 
25 the first -order eigencomponent. 
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In step #30, using the difference obtained in step #29, a second order K- 
L coefTlclent C/^^^ (which is also called second order eigencomponent) is 
calculated using equations (4), (6), (7), (14) and (16). The second order K-L 
coefficient C/^^^ is used in the test mode in K-L conversion (step #40). 

The test mode starts from step #32 and continues to step #42. The test 
mode is provided to generate first order component feature W^^^ and second 
order component feature W^^^ 

In step #32, a face to be tested is input. 

In step #33, the input face image is divided into a plurality of facial parts, 
such as right eye, left eye, right eyebrow, left eyebrow, nose, mouth, facial 
configuration, and each part is analyzed to obtain basic facial component . 
The facial component O,. can be weighted according to the significance of 
identity of human face. 

In step #34, the basic facial component is subtracted by the first order 
average facial component ^ to obtain a first difference r^^^ (also called a first 
order residue component). The first difference r^*^ is applied to step #35 and to 
step #39. 

Steps #32 to #34 taken together is called an analyzing step for analyzing 
the test face image. 

In step #35, using the first difference r^^^ and the first order 
eigencomponents U^^^ . K-L conversion is carried out by the use of equation (9) 
to obtain the first order component feature 

In step #36, the first order component feature W^^ is produced. The first 
order component feature W^^^ represents the feature of the test face input in 
step #32. The first order component feature W^^^ can be used as information 
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representing the test face, but has a relatively large size of data. A further 
calculation is carried out to reduce the data size. 

In step #37, using the equation (11), K-L inverse conversion is carried 
out to generate a reconstructed matrix f^'^ . 
5 In step #38, the reconstructed matrix f^^^ is produced. 

In step #39, the equation (12) is carried out to produce a difference 

between the first difference r^^^ and the reconstructed matrix f^*^ to obtain a 
second difference F^^^ , which is generally referred to as a second residue 
component r/^^ . 

1 0 Steps #37 to #39 taken together is called an analyzing step for analyzing 

the first -order component feature. 

In step #40, using the second difference r^^^ and the second order K-L 

coefficient U^^^ , K-L conversion is carried out by equation (17) to generate the 

second order component feature W^^^ . 
15 In step #41 , the second order component feature W^'^^ is produced. The 

second order component feature W^^^ has information representing the test face 

as entered in the test mode. 

It is to be noted that the flowcharts shown in Figs. 3 and 4 can be 

arranged by a computer connected with a camera for capturing the sample face 
20 images and the test face image. It is possible to prepare two sets of 

arrangements, one set is for the training mode operation, and another set for 

the test mode operation. Each set includes a computer and a camera. The set 

for the training mode operation is programmed to carry out steps #22-#30, and 

the set for the test mode operation is programmed to carry out steps #32-#42. 
25 In the set for the test mode operation, a memory is provided to previously store 
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information obtained by the set for the training mode operation, such as the first 
order average facial component ^ , the first order eigencomponents U^^^ , the 
inverse matrix U^^^ , and the second order eigencomponent U^^^ . 

This invention is very effective for describing human faces using 
component-based features. Since the high-order eigencomponents can be 
calculated only once with the training components, the high-order component 
features can be obtained as efficient as first-order component features. 
However, since detailed regional identity information can be revealed with high- 
order component features, the combination of first-order component features 
and high-order component features of eyes, eyebrows, nose, mouth and outline 
with different attention weights has better face description capability compared 
with the first -order eigenface features or combined first -order and high-order 
eigenface features. 

This Invention Is very effective and efficient in describe human faces 
which can be used in internet multimedia database retrieval, video editing, 
digital library, surveillance and tracking, and other applications using face 
recognition and verification broadly. 



